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1. Abstract 


UNISON was an Investing in Innovation (13) development grant funded by the Office of Innovation 
and Improvement, U.S. Department of Education to Take Stock in Children (TSIC), a state-wide, 
501(c)(3) non-profit, private-public partnership in Florida. UNISON provided a whole school 
mentoring approach and focused on developing students’ non-cognitive skills with a focus on 
improving academic success and attainment thereby impacting graduation rates in low-performing 
schools. Intervention strategies included a whole-school culture program with clubs and activities 
focused on sustaining the culture of kindness and compassion, BARR teacher training, block 
meetings, and implementation of the I-Time curricula, TSIC 1:1 mentoring, group mentoring, 
summer Leadership Institute, and parent engagement. The grant funded a collective impact 
coordinator for each county and a college success coach in each school. UNISON targeted three 
treatment schools in northeast Florida, and the impact study included 15 comparison schools using 
a comparative short interrupted time series (CSITS) to measure the effect on graduation rates after 
three years of intervention strategies. Using the publicly available graduation rate data from the 
Florida Department of Education website, baseline equivalence was established based on 
rural/urban county designations and baseline graduation rates from 2014-15 prior to the 
intervention. The study found a statistically significant positive difference in the one urban and 
two rural treatment schools compared to the six urban and nine rural “business-as-usual” 
comparison schools. The treatment schools gained almost 20 percentage points in their graduation 
rate during the three-year intervention from 72.43% to 91.73%, while comparison schools moved 
from 72.98% to 83.40%. 
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2. Introduction 


UNISON: Uplifting Non-cognitive Skills and Innovation through Student Opportunity Networks, 
a 4-year i3 development project to address Absolute Priority 2: Improving Low Performing 
Schools, Subpart B: Implementing programs, supports, or other strategies that improve students’ 
non-cognitive abilities, was awarded to Take Stock in Children (TSIC), a state-wide, 501(c)(3) 
non-profit, private-public partnership in Florida. UNISON focused on two goals: 1) build strong 
evidence of success for promoting students’ non-cognitive skill set development to improve 
academic achievement; and 2) strengthen the case for adoption of a collective impact, whole- 
school mentoring approach as an effective vehicle to increase stakeholder engagement, promoting 
sustained improvements in low-performing schools. 


Research has suggested students’ non-cognitive skills are a key component of their academic 
achievement, persistence, and secondary and post-secondary attainment (Conley, 2007; Dweck, 
Walton, & Cohen, 2011; Farrington, et. al, 2012; Heckman, 2008), but low-income and minority 
students often lag behind their non-minority, more affluent peers in developing these skills 
(Farrington, et. al, 2012; Heckman, 2008). While non-cognitive skills are typically fostered in the 
home environment, outside forces (e.g. social capital, school factors) including mentoring, can 
further their development, particularly in high-need students (Lippman, Atienza, Rivers, & Keith, 
2008; Wimberly, 2002). Existing research has been correlational and it is unclear which particular 
factors can be intentionally developed in classroom or whole school contexts or if changing them 
impacts academic performance (Farrington, et. al, 2012; Heckman, 2008). 


Previous research on the TSIC traditional mentoring model and College Success program indicates 
their potential to increase high school graduation and college enrollment rates. This is evidenced 
by substantially higher graduation and college enrollment rates among TSIC Scholars than peer 
comparison groups and statewide averages, in student populations that are predominantly minority 
(62%) and economically disadvantaged (100%) (The Evaluation Group, 2013). 


The UNISON project used three unique elements that differed from previous studies including: 1) 
schoolwide mentoring at the whole-school, classroom/small group, and individual levels to build 
high school students’ non-cognitive skill sets and improve academic achievement; 2) building 
students’ self-efficacy to successfully access post-secondary education and career goals within the 
context of a whole-school mentoring approach; and 3) aligning existing and new school and 
community supports around a shared impact agenda to create and promote sustained improvements 
in low-performing schools. The combination of these three elements and their impact on the 
development of non-cognitive skills, academic achievement, and post-secondary access in three 
differing contexts (i.e. whole-school, classroom, individual levels) in three different settings (1.e. 
urban, suburban, rural) made the project approach novel compared to previous studies. 


2.1 Description of the Intervention 


UNISON implemented key program components at the school level, the classroom level, and the 

individual student level. While several interventions were selected in the proposal, other strategies 

were finalized during the first year such as the culture program. This framework of strategies 

represents key focal areas based on research and prior successes from the first TSIC 13 FLIGHT 

(The Evaluation Group, 2013; The Evaluation Group, 2014), thus future replication allows schools 
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to select curricula or programs relevant to address their students’needs. Each strategy includes a 
description as well as information about implementation. 


1. Collective Impact 
a. The Collective Impact Teams were formed in 2015 and developed comprehensive written 
plans describing the collective impact process, strengths, challenges, goals, and road maps for 
each school. Collective Impact Team meetings were held monthly throughout the grant period. 
Teams created and implemented new initiatives centered around increasing students’ social 
and emotional learning, promoting and advising students on college and career readiness, and 
creating a college-going culture within the schools. The teams’ activities included 
collaboration on school college and career fairs, coordination of school visits from college and 
university representatives, and coordination of college tours and FAFSA workshops for 
seniors. They also initiated bi-monthly visits with college advisors to the high school 
campuses. Students and parents scheduled on-site appointments at each high school for 
advisement on financial aid, college enrollment, dual enrollment, career opportunities, and 
college programs. 


2. School Culture 
a. Rachel’s Challenge 

i. Principals selected Rachel’s Challenge (RC) as the program they thought would 
promote a positive and immediate affect on their school culture. RC is based on the 
writings and life of 17-year-old Rachel Scott who was the first student killed at 
Columbine High School in 1999. RC is a school-wide intervention designed to create a 
culture of kindness and compassion and equip students and adults with the skills 
necessary to create and sustain a safe, caring, and supportive learning environment. The 
elements of RC included a 1-hour PD session delivered to the entire school staff that 
was was implemented in Year | only; a kick-off assembly delivered to all students and 
staff that was implemented in Years | and 2; and a Legacy assembly offered to students 
in Year 2 who had previously attended the kick-off assembly in Year 1. During Years 
3 and 4, schools selected a whole-school culture presentation/presenter to transition the 
school from using RC and Rachel’s Legacy to continue the kindness theme, but at the 
same time to take ownership of their school culture and climate for sustaining the work. 

b. School Culture Clubs 

i. Friends of Rachel (FOR) Student Clubs were formed and were intended to reinforce the 
RC message and create a permanent and positive cultural change in the school through 
acts of charity, appreciation, kindness, and compassion. Club members held monthly 
meetings to build cohesiveness, define roles, and devise strategy. They then conducted 
activities that reinforced the school-wide message of showing empathy and compassion 
to others. Club meetings and activities were implemented all four years. 

ii. Towards the end of the Year 2, the FOR Clubs transitioned to a more sustainable 
initiative that reflected the spirit and culture of each school. Both Columbia High 
School and Andrew Jackson High School share the same school mascot, the Tigers. 
Students at both schools changed the FOR Club name to The Pawsitives Club as a 
reflection of their schools’ identity and culture. Fort White High School incorporated 
the FOR Club mission and activities into existing school clubs. 

c. Teacher Professional Development Training 
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i. Teachers and school staff were provided on-site trainings by experts to increase 
awareness and understanding of social and emotional learning, noncognitive skills 
acquisition, and trauma-informed care for students. Teachers learned how social and 
emotional skills can positively impact students’ learning, motivation and persistence. 
Teachers were given tools and activities to reinforce and incorporate the lessons into a 
classroom setting. Teachers also learned that, by fostering better relationships with 
students, they can increase student engagement and learning which will also result in a 
decrease of classroom disruptions. 


3. BARR 

a. Professional Development: 9th grade and 10th grade teachers were provided with two days 
of professional development in recognizing and growing student’s non-cognitive skills. In 
Years 2 and 3, teachers received an additional six hours each year of BARR training on 
relevant topics to continue learning about social-emotional development and non-cognitive 
strategies. BARR trainers conducted site visits each year to provide support to the school 
teams. 

b. I-Time: Teachers conducted weekly group meetings with 9th grade students to strengthen 
and address non-cognitive factors. The I-Time curriculum consists of 33 structured 30- 
minute activities in the following 10 areas: Building a Connected Community; Setting 
Goals; Leadership; Communication; Understanding your Assets; Grief and Loss; Bullying; 
Diversity; Risky Behaviors; and Pursuing your Dreams. 

c. Student Risk Management: Classes in the 9th grade were structured into blocks of four 
teachers, (social studies, English, math and science) with approximately 85 students per 
block. Teachers met weekly in Block meetings to discuss the progress of each student and 
provided real time interventions to support to students that were falling behind. High-risk 
students were referred to a risk review team for additional services. 


4. College and Career Readiness 
a. TSIC college and career readiness advocacy services 

i. TSIC college and career readiness advocacy services were provided to increase 
academic achievement and students’ readiness for post-secondary education and 
employment. We provided the services on a schoolwide basis for the last three years 
(2017-2019), with a significant concentration on the senior class at all three schools. 
The staff, in partnership with our Collective Impact Teams, were creative in how they 
delivered the services in order to reach as many seniors as possible. Staff worked with 
the senior teachers to provide classroom or small group presentations on important 
topics such as financial literacy, essay writing, FAFSA completions, scholarship 
applications, and ACT/SAT prep. They also invited local businesses to present career 
opportunities to students and the associated post-secondary pathways that are available 
to them at local colleges. Seniors at all three high schools completed post-secondary 
plans, participated in scholarship boot camps, and college representatives met with 
students and parents throughout the year on the high school campuses. All students 
were invited to attend workshops and participate in various activities that focused on 
college readiness. 

b. Parent Engagement 

i. School phone banks, a texting platform and social media were employed to increase 

parents’ awareness of the college enrollment process and offer assistance to parents so 
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that they could provide support, materials and guidance to their students. Parent 
Orientations, Ed Fairs and College and Career Fairs were also used as opportunities to 
educate parents. A Senior Parent Handbook/folder was distributed to parents that 
contained a parent and student college readiness checklist; high school graduation 
requirements; deadlines for college applications; FAFSA, financial and scholarship 
information; and a list of college readiness websites. The folder also contained a letter 
from the local college outlining the financial aid opportunities and affirming the 
affordability of attending college. College representatives also visited high school 
campuses twice a month to meet with students and parents to help with college 
applications, FAFSA, dual enrollment, and financial aid and staff was available to 
parents on an individual basis to answer questions on how to select the right college, 
explain the application process and give guidance on financial aid. 


5. Mentoring 

a. Traditional mentoring: UNISON utilized the already successful TSIC school-based 
mentoring model using a traditional 1:1 match with eligible students per school. Students 
met weekly with their mentors for 30-45 minutes during the school day, often over lunch. 
Students were provided with a college success coach and a TSIC Florida Prepaid STARS 
scholarship if they actively participated in the mentoring program; maintained grades; 
avoided suspensions, expulsions, or criminal activity; and successfully completed high 
school. Scholarships were not paid through i3 funds. 

b. Non-traditional group mentoring: UNJSON implemented a new non-traditional mentoring 
approach which provided targeted group mentoring (in which school faculty and staff or a 
college/career/ community volunteer mentor student groups) for students that met the same 
eligibility guidelines as a TSIC student. Groups focused on college and career readiness 
and promoting social and emotional learning skills such as self-awareness, social 
awareness, self-management, relationship building and responsible decision making which 
encouraged motivation, persistence and conflict resolution skills. 

c. Peer-to-peer mentoring: UNISON introduced peer mentoring to the schools in 2017-18. 
Upperclassmen, who were typically members of the school culture clubs, took leadership 
roles in presenting their high school experiences, lessons learned and the importance of 
graduating high school to the underclassmen. They also served as school ambassadors and 
peer mentors to students who needed assistance and encouragement. 

d. Summer Leadership Institute: A day-long Summer Leadership Institute was offered to 75 
or more students in three successive summers. The purpose of the institute was to provide 
leadership training, communication skills and further prepare students for college and 
career through supplemental college prep workshops. During the third year, the program 
was offered while school was still in session to minimize challenges of time and 
transportation during the summer. 


In addition to the strategies, UNJSON embedded a TSIC college success coach in each school and 
a collective impact coordinator in each county to facilitate delivery of BARR, RC, and mentoring; 
coordinate school-based and community-based resources; coordinate college tours or college 
representatives on campus; invite guest speakers, and provide student workshops, advocacy, 
support, and PD using the TSIC model. 
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2.2 Logic Model 


The UNISON logic model (Figure 1) shows our theory of change based upon the three grouping 
levels (school, classroom, and individual) with the key components and planned activities 
associated with each level. Additionally, the linkage from activities to short-term or mediating 
outcomes that, in turn, impact the long-term outcome/impact of increased graduation rates. 


Vision: UNISON will expand TSIC’s traditional 1:1 mentoring model to implement a novel, non- 
traditional schoolwide mentoring approach designed to build students’ non-cognitive skill sets 
utilizing three strategic pathways at the whole school, classroom, and individual level. 


Goal 1: Improve students’ non-cognitive skill sets to increase academic achievement and students’ 
readiness for post-secondary education and employment. 


Goal 2: Strengthen the case for adoption of a collective impact, whole-school mentoring approach 
as an effective vehicle to increase stakeholder engagement, promoting sustained improvements in 
low-performing schools. 
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Figure 1. UNISON Logic Model 


RESOURCES ACTIVITIES MEDIATING OUTCOMES IMPACT 
e Mentor Corps 1. Collective Impact Team 
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Impact Teams = improvement plans e Decrease discipline referrals 
e College e | 2. Culture Program e Coordinate, systemic post- 
Success Pe Implement Rachel’s’ Challenge (RC), a school-wide social-emotional program secondary ae 5 
Coach at each is designed to improve the school culture by promoting empathy and compassion by: e Complete strategic plan 
school and : e Orienting teachers to RC (Year | only) e CIT collaboration assessment 
Collective = | e Presenting the principles of RC in an inspirational school-wide assembly e CIT monthly meetings and 
Impact e Holding monthly Friends of Rachel (FOR) Club meetings to plan sustaining events referrals/coordination of services 
Coordinator e Conducting FOR activities that build and sustain the concepts of empathy and e Increase parent engagement N 
for each compassion = 
e TSIC College ss pel: ees ene Risks eee ; Aare eet ° rie academic achievement 2 
mplement the non-cognitive skills development program with participating ae ; 
pe a g and 10" grade teachers and students by: : oe alee nave ae = 
model A e Providing 2-day PD to teachers (Year 1; supplemented with six hours in Years 2 & Sa ee or S 
¢ School district | & | 3) 
ee . 2 Ie Delivering I-Time curriculum to 9" grade students y ae ee eee S 
three high e Student Risk Management teacher meetings e ey equip ee : iol) 
schools 4. College and Career Readiness o 
e Existing e TSIC college and career readiness advocacy services a 
programs at : ; 
caahechaal e Parent engagement using phone banks, meetings 2 
sidand pia 5. Mentoring . . 7 e Increase college and career S 
furiding & Implement quality mentoring and leadership opportunities to promote college and aspirations Rea aa 
SCoannunigy = | career readiness with at-risk students by: e Increase ACT/SAT participation 
Partners = e Providing traditional 1:1 mentoring, college scholarships, and employment rate and scores 
° Colleges & = opportunities to eligible students in each school e Increase FAFSA completion rate 
Universities = | Implementing a non-traditional group mentoring approach to students that meet the _| ¢ Increase application submissions 


same eligibility requirement in each school 

e Peer-to-peer mentoring using upperclassmen or club leaders to talk to underclassmen 
about high school graduation requirements and college and career readiness 

e Reinforce college enrollment, leadership, and communication skills through a 
Student Summer/Yearend Leadership Institute 


to college or for 
scholarships/financial aid 
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3. Impact Study Design 


UNISON used a comparative short interrupted time series (CSITS) quasi-experimental design 
(QED). The three treatment high schools were matched to comparison schools at a 1:5 ratio using 
scaled Euclidean distance scores without replacement to ensure baseline equivalence on the key 
school characteristic of graduation rate in the year immediately prior to the intervention (2014- 
15). Schools’ graduation rates were tracked for eight years over the course of the time series, 
starting five years prior to the start of the UNISON intervention and ending three years after the 
start of the intervention. The impact study used the federal 4-year adjusted cohort graduation rate. 
Federal regulations require each state to calculate a 4-year adjusted cohort graduation rate, which 
includes standard diplomas but excludes GEDs, both regular and adult, and special diplomas. The 
US Department of Education (USED) adopted this calculation method to develop uniform, 
accurate and comparable graduation rates across all states. 


The USED required states to begin calculating the new graduation rate in 2010-11. The 4-year 
adjusted cohort graduation rate is the percentage of students who graduated with a standard 
diploma within four years of their initial enrollment in ninth grade (the numerator). Incoming 
transfer students were included in the appropriate cohort based on their grade level and year of 
entry. Deceased students and students who withdrew to attend school in another school system 
that would culminate with a standard diploma were removed from the cohort (the denominator). 
Each student in the resulting adjusted cohort received a final classification as a graduate, dropout 
or non-graduate. The federal 4-year adjusted cohort graduation rate was used for the five pre- 
treatment years (2010-11, 2011-12, 2012-13, 2013-14, 2014-15) and three treatment years (2015- 
16, 2016-17, and 2017-18). Data collected by the evaluation team came from publicly available 
graduation rates published annually by the Florida Department of Education. 


SY2015-16 was the first year of the UNJSON intervention. Graduation rates at the end of the 
SY2016, SY2017, and SY2018 included in the impact study assessed program impact after 
implementation for one to three years. Each treatment school year contributed a data point to the 
treatment time series. Table 1 below indicates the five pre-treatment years (SY 2010-11 to 2014- 
15) and the three treatment years (SY2015-16 to SY 2017-18) for UNISON and comparison 
schools. 


Table 1. Pre-treatment Years and Treatment Years for Assessing Graduation Rate in 


UNISON and Comparison Schools 
School Years 


2010-11 | 2011-12 | 2012-13 | 2013-14 2014-15 | 2015-16 | 2016-17 | 2017-18 | Number of 
Schools 
UNISON 
x x x x x T T T 3 
Schools 
Comparison 
P x x x x x t t t 15 
Schools 


Notes: Graduation rates are compiled at the conclusion of each academic year and reported in the fall. 

“x”: indicates a pre-treatment year when a school-level graduation rate will be obtained 

“T”: For Treatment schools T indicates a treatment year 

“t”: For comparison schools, “t” indicates a year when the schools’ treatment group counterparts have received 


treatment 
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3.1 Sample 


UNISON proposed to test the intervention in one school located in an urban, suburban, and rural 
locale. In the first academic year of the intervention (2015-16), one of the schools became a 
dedicated military leadership academy. The decision was made to replace this school, and the 
external evaluator conducted a thorough analysis to identify a replacement school that met the 
criteria for the proposed intervention (i.e., Title 1, serving high-need students). This school was 
replaced by a rural school that began full implementation of all strategies in the 2016-17 school 
year. The three treatment schools were from Columbia County (n=2) and Duval County (n=1). 
Given the change in schools to include two schools from Columbia County, both schools were, in 
effect, rural rather than one rural and one suburban; thus, the analysis focused on comparing 
graduation rates based on the county designation for rural or urban according to the 2010 Census. 
The two schools from Columbia County represented rural while the one school from Duval County 
represented an urban setting. 


“Business-as-usual” schools did not participate in the UNISON program. Scaled Euclidean 
distance scores were used to create a comparison set of 15 schools. The three treatment schools 
come from two school districts; the 15 comparison schools came from 12 county-level districts in 
Florida. 


The evaluation team reviewed the list of potential comparison schools and removed the following: 
school for the deaf and blind, virtual/online schools, detention, residential, exceptional, hospital or 
jail, and career. We also removed schools containing any of the following key words: middle, 
junior, youth, center, or academy. Finally, we removed schools with a 0% or 100% graduation 
rate. The final dataset contained 533 potential comparison public schools for matching with the 
three treatment schools. A total of 543 schools were removed prior to running the propensity score 
match (PSM) for comparison schools, leaving 533 schools to identify 15 matches. 


As stated previously, our program took place in one urban school and two rural schools. An initial 
PSM run using the MatchlIt program in R, included key independent variables such as graduation 
rate, urbanicity, percent non-white, and average daily attendance. Our limited sample size resulted 
in an underpowered model; therefore, we removed variables one at a time, but in the end, used 
graduation rate (which is our outcome or dependent variable) to establish baseline equivalence. 


Further limitations from the initial PSM resulted in the selection of 15 comparison schools to the 
comparison group that were closely matched to the schools in the treatment group with a baseline 
difference of 0.05 measured with Hedge’s g. However, only two rural schools were pulled to match 
with our two rural treatment schools. Our proposed ratio of 1:5 was not met using PSM. In In 
accordance with What Works Clearinghouse (WWC) 4.0 guidelines (WWC, 2017), we stratified 
the matching by urbanicity. This resulted in 62 potential schools in the rural stratum and 471 
potential schools in the urban stratum. 


We ordered the list of schools within the rural and the urban strata by graduation rate and selected 
the nine comparison schools closest to the rural treatment schools within the rural stratum and six 
comparison schools from the urban stratum closest to the urban treatment school. Checking for 
complete data revealed one comparison school was missing the first pre-treatment year graduation 


THE EVALUATION GROUP: PAGE 12 OF 22 


rate and one school was replaced missing the final year of data. These schools were replaced and 
the Hedge’s g recalculated. 


3.2 Study Question 


There was one main confirmatory research question: What is the effect of the UNJSON program 
on school-level graduation rates for 12" grade student cohorts after three years of the intervention, 
as compared to school-level graduation rates for 12" grade student cohorts in the “business-as- 
usual” condition? 


4. Analysis and Results 


4.1 Baseline Equivalence 


Baseline equivalence was established for the final set of 15 comparison schools and three treatment 
schools and resulted in a difference measured by Hedge’s g of -0.21, thereby meeting baseline 
equivalence on graduation rate with statistical control. Our planned comparison called for a 1:5 
comparison school selection using urbanicity as a stratification variable and a covariate in the 
planned impact analysis for the UNISON intervention; however, our baseline sample included six 
urban and nine rural schools. Our comparison data included in Table 2 satisfies the two conditions 
in the What Works Clearinghouse Standards 4.0 (p. 15). First, baseline and outcome data came 
from the same measure since we used the publicly available graduation rate. Second, there is a 
correlation between the baseline and outcome measure of .60 or higher, which can be seen in Table 
2 for at least two time periods. 


Table 2. Correlation of Graduation Rates for Baseline and Outcome Years 


2015-16 | 0.076] 0.323| 0.241] 0.456] 0.061] | 0.09] _0.085 
2016-17 | 0.202] 0.3 | 0.23] 0.025] 0.264] 0.09] 572% | 


Note: **p < .01 *p<.05 


The means and standard deviations of the treatment and comparison schools are listed below for 
the matching variable, graduation rate, from SY2014-15: 


e Treatment (n=3): Mean 72.43, SD 1.85 
e Comparison (n=15) Mean 72.98, SD 2.63 
e Hedge’s g = -0.2072 


Table 3 includes characteristics for all schools accessed from the Florida Department of Education 
website (FLDOE, 2015a-c) for baseline year 2014-15. 
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Table 3. 2014-15 Baseline Demographic Data from Treatment and Comparison Schools 


Janell Percent Average 
Urban Number Mle Graduation 
School Treatment Males | Females non- daily 
County Enrolled : rate 
Tne white attendance 
Andrew ian 
Jackson Yes Duval 731 339 392 97% 95.1% 70.3% 
High School 
Columbia Rural ; és 5 
High School Yes Célacabia 1,913 945 968 34% 93.1% 73.4% 
Fort White 
: Rural 
High Yes . 607 308 299 21% 92.5% 73.6% 
Columbia 
School* 
Anclote Urban ‘ a é 
High School No a, 1,382 724 658 31% 92.9% 69.9% 
Hudson : Urban . a é 
Hich Schiéol No Pasca 1,267 622 645 16% 91.7% 70.9% 
Chamberlain ; Urban a i A 
High School No Halisborough 1,772 937 835 76% 90.7% 69.1% 
Miami . F Urban 
Senior High No Dade 2,964 1,583 1,381 98% 91.4% 70.5% 
School 
Port St. eben 
Lucie High No : 1,561 838 723 51% 88.6% 70.7% 
St. Lucie 
School 
T. Dewitt Liebe 
Taylor High No . 576 302 274 60% 93.0% 70.6% 
Volusia 
School* 
Clewiston : Rural rn é é 
High School No Hendry 937 460 477 78% 92.1% 76% 
Cottondale Ruial 
High No 1,518 761 757 90% 91.7% 76.4% 
Jackson 
School* 
Graceville Rural 
High No 237 118 119 33% 94.3% 715% 
Jackson 
School* 
Hamilton Rural 
High No ua 420 215 205 53% 90.6% 75.3% 
Hamilton 
School* 
Marathon Rural F si Pe 
School* No snene 404 210 194 53% 93.2% 74.1% 
Mariana Rural 3 é é 
High School No feckcca 779 354 425 42% 90.7% 76.9% 
Okeechobee Rural . 4 & 
High School No Gikeechabes 1,751 923 828 46% 91.4% 71.4% 
Faylon Rural 
County No Tavior 662 335 327 30% 90.8% 73.9% 
High School y 
West 
Gadsden Rural ; ‘ " 
High No Gaicden 235 115 120 93% 94.5% 74% 
School* 
Treatment 
Average Yes Rural 1,260 627 633 27% 92.8% 73.5% 
n=2 
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Table 3. 2014-15 Baseline Demographic Data from Treatment and Comparison Schools 


Rural’ Percent Average 
Urban Number : 8 Graduation 
School Treatment Males | Females non- daily 
County Enrolled : rate 
white attendance 

name 
Comparison 
Average No Rural 706 361 345 59% 92.8% 74.8% 
n=9 
Treatment 
Average Yes Urban 731 339 392 97% 95.1% 70.3% 
n=! 
Comparison 
Average No Urban 1,587 834 753 56% 91.4% 70.3% 
n=6 


Figure 2 shows the locations of treatment schools within county designations of rural or urban 
based on the 2010 census data. Figure 3 includes all treatment and comparison schools with rural 
counties shaded blue and urban counties shaded orange. Each county includes the number of 
schools participating in the study. 


Figure 2. Treatment Schools by County Designation. 


UNISON Number of Schools for Treatment Group 


vy Bi + 
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Figure 3. Rural and Urban Counties and Number of Schools per County. 


Rural Urban 


UNISON Number of Schools by Counties BB Rural 


BB Urban 
Q 


vy Plt 


4.2 Confirmatory Analytic Model 


This impact study tested a bundled intervention. In one key component, student mentoring, Florida 
pre-paid scholarships were provided to each of the students enrolled in the traditional 1:1 TSIC 
mentoring program. These scholarships were offered by TSIC, the developer of the i3-funded 


intervention, but were paid for privately and are NOT paid through 13 funds. This design was 
unconfounded. 


The model specified below is a two-level hierarchical linear model with repeated observations 
over time nested in schools!. The subscripts i and j represent the i" time point in the /“ school. The 


dependent variable (Gi) is the graduation rate at the i" time point in the j school. The model 
includes random intercepts for schools (denoted as a ), and a residual error term (denoted as 


Years 


ei 


), and indicator variables for treatment school, treatment years, and a treatment school-by- 


treatment year interaction term. The impact model will include an adjustment for potential 
autocorrelation among repeated observations within schools over time. 


Model (2-Level Model Written in Combined Notation) 


Gi= Bor BT) + Bo (TrtYri) + f(T; * Trt¥ri) + 2 4RuralStatus + wi" +25" 


if 


' For a detailed explanation of the C-SITS design, See Price, C. (2013). Research on Educational Effectiveness 
(SREE) 2013 Spring Conference Workshop: Planning for a Short-Interrupted Time Series Design. Workshop 
materials updated 03-30-2015. 
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Years = 


the graduation rate from the i™ time point in the j® school 
1 if school j is an intervention (treatment) school, and 
0 if comparison school 
1 if year is a treatment year (indicated by “T” or “t” in Table 1 below) 
0 if a pre-treatment year (indicated by “x” in Table 1), above. 
the intercept, which is the comparison school mean graduation rate in pre-treatment 
years for schools. 
is the deviation of school j’s intercept from the mean intercept, conditional on Tj, 


distributed with mean 0 and variance o%,,, 


the average difference between treatment and comparison schools during pre-treatment 
years; 

the average difference between pre-treatment years and treatment years for comparison 
schools. (Comparison schools are never treated, but the “treatment years” are the years 
indicated with lower case “t” in Table 1. They are the years in which the treatment 
schools that they are matched to are being treated.) 

the treatment effect. This is the difference-in-difference estimator. It is the difference 
between treatment and comparison schools in their differences between pre-treatment 
and treatment years. 

The effect of rural status, where | indicates a rural school district and 0 indicates an 
urban school district. 

the random error effect representing the difference between the graduation rate at year i 
for school j and the predicted graduation rate for school j. These residual effects are 


assumed normally distributed with mean 0 and variance oy,,,,, and are assumed to 


Schs 


have 1“ order autoregressive correlation. They are assumed independent from y; 


The CSITS design used five years of pre-treatment data and three years of treatment data. UNISON 
was a 4-year program, ending on 3/31/19, however, only three years of treatment data were used 
in the impact study. Graduation rates on the previous school year were compiled over the summer 
and reported in the fall of the next school year. Our final data for SY2017-18 was publicly available 
as of December 2018. 


Table 4. Pre-treatment Years and Treatment Years for Assessing Graduation Rate in 


UNISON and Comparison Schools, with Coding of ntime and TrtYr. 


School Year 
2010-11 | 2011-12 | 2012-13 | 2013-14} 2014-15 | 2015-16 | 2016-17} 2017-18 | Number of 
Schools 
UNISON x x x x x T T T 3 
Schools 
Comparison 
Schools x x x x x t t t 15 
ntime coded as: -4 -3 -2 =] 0 1 
TrtYr coded as: 0 0 0 0 0 1 1 1 


665,99, 


treatment. 


Graduation rates are compiled at the conclusion of each academic year and reported in the fall. 
x”: indicates a pre-treatment year when a school-level graduation rate will be obtained 
“T”: For Treatment schools T indicates a treatment year 
“t”: For comparison schools, “t” indicates a year when the schools’ treatment group counterparts have received 
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Florida graduation rates were compiled annually to meet federally mandated reporting 
requirements and were publicly available on the FLDOE website. Upon reviewing the final list of 
selected schools identified in our baseline equivalence test (see next section) to determine if all 
eight years of data was available, one school was replaced as the first pre-treatment year was 
missing. Additionally, a rural school missing the final year official graduation rate was also 
replaced so that all schools had no missing data for the eight-year span. 


4.3 Results for Confirmatory Question: Graduation Rate 


The CSITS analysis utilized a difference-in-difference model to test for mean differences in 
graduation rates of treated schools compared to comparison schools. The graduation rate from the 
five pre-treatment years were all coded as a 0 and the three treatment years were coded as 1. The 
difference-in-difference compares the average mean score from the three treatment years to the 
five pre-treatment years. The final mean average (SY2017-18) for the treatment schools was 91.73 
(2.41) and the final mean average for the comparison schools was 83.40 (6.52). Baseline for all 
schools from 2014-15 was 72.43 (1.85) for treatment and 72.98 (2.63) for comparison schools. 
The analysis found a statistically significant result for treatment schools for the variable “time by 
condition” (treatment years times school condition (0 for comparison and | for treated)) (t-value 
2.1859, p = 0.0307). A summary table of the regression is included in Table 5. 


Table 5. UNISON Graduation Rate Model 


Variable Estimate Standard Error t-value p-value 
Intercept Pee 48.05 0.67 0.503 
Baseline 0.54 0.68 0.79 0.442 
Condition -3.43 2.48 -1.38 0.188 
TrtYr 8.39 137 6.13 0.000*** 
TimeByCondition | 7.30 3.35 2.18 0.03 1* 
RuralStatus -0.55 ao -0.16 0.872 


Note: ***p < .000; **p < .01; *p < .05 


Figure 4 shows the annual means for treatment and comparison schools. Data points are listed by 
the end of year for each school year beginning with 2010-11 and ending with 2017-18. Baseline is 
noted on the graph for 2014-15 with the three intervention years showing an increase above the 
comparison group by the end of the three-year intervention. The average for treatment and 
comparison schools are listed below in Table 6 with a row to show the difference in mean averages 
for each of the eight years. 
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Figure 4. Annual Graduation Rates 2010-11 through 2017-18 
91.73 ecm 


83.40 


Baseline graduation rate 2014-15 


Avg, Graduation Rate 


Table 6. Mean Average Graduation Rate for Treatment and Comparison SchoolsThe a 
Condition Pre-Treatment Years Treatment Years 


Year 2010-11 | 2011-12 | 2012-13 | 2013-14 | 2014-15 | 2015-16 | 2016-17 | 2017-18 
Treatment 63.07 66.95 67.42 68.00 72.43 78.13 79.47 91.73 
Comparison | 70.17 71.29 71.40 72.05 72.98 77.09 79.20 83.40 


ea 7.10 | 434 | 3.98 | 4.05 | -0.55 1.04 0.27 8.33 


5. Discussion 


Prior to the intervention, Florida Department of Education released their Every Student Succeeds 
Act (ESSA) plan to increase the 4-year adjusted cohort graduation rate by 7.1 percentage points 
from 77.9 in 2014-15 to 85% by 2019-20. The three treatment schools surpassed this rate during 
the 2017-18 school year by an additional 6.73 percentage points, while the comparison schools fell 
short by 1.60 percentage points. 


A difference-in-difference means regression conducted in R found significance for our treatment 
variable and the interaction variable ‘TimeByCondition’. The three treatment schools gained 19- 
percentage points over the 2014-15 baseline in the three years during the intervention compared 
to a 10-percentage point gain by the 15 comparison schools. Through a stratified propensity score 
matching, equivalence was established using our outcome variable, graduation rate, as well as 
urbanicity, thus ensuring that our results supported the intervention as one contributing factor to 
the increase in graduation rate. As seen in Table 2, the percent non-White and average daily 
attendance were not included as co-variates in our baseline model due to our sample size and 
proposed research question. We recognize the limitations in the differences in ethnicity, 
specifically between urban and rural schools and treatment and comparison schools. The ideal 
study could include these and other variables that potentially impact graduation rates. While all 
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schools by virtue of the ESSA plan reported gains in graduation, the use of the CSITS design takes 
into account the average growth both pre- and post-intervention. 


While the intervention was delivered as a bundle during the three years, during year four of the 
grant, the schools adapted/sustained several key strategies, including the whole-school culture 
assembly, the clubs and activities, and the college and career-ready activities. In the two Columbia 
County high schools, the weekly I-Time sessions were restructured into a freshmen Workplace 
Essentials required course, teachers trained in BARR continued to create classroom cultures of 
trust and respect focusing on building relationships with their students and among their students, 
and TSIC 1:1 mentoring continued for eligible students, as well as group mentoring contingent on 
available school or community mentors. 


As noted in our introduction, the proposed implementation sites included a suburban/city fringe 
high school. Due to the change in schools in Year 1, the resulting analysis occurred in one urban 
and two rural schools, thus eliminating our ability to assess three different school settings as a co- 
variate of graduation rate. While all schools reported gains in graduation rates compared to 
baseline, the three treatment schools, exceeded the Florida ESSA 2020 goal by an additional six 
percentage points suggesting that the UNISON program positively impacted student graduation 
rates through the implementation of key strategies. Thus, further testing the implementation of key 
strategies by other schools within or outside Florida could prove the success of these strategies as 
influencing the graduation rate for students. Additionally, increasing the sample size of treatment 
and comparison schools would allow for subgroup analyses to disaggregate the findings by gender, 
race/ethnicity, economic disadvantage status, English language learners, and students with 
disabilities. 
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